Automated Venting and Refilling of Multiple Liquid Cooling Systems

ABSTRACT

A two-phase liquid cooling system includes an active venting system for regulating an amount of non-condensable gas within the cooling system. Various venting structures may be used to remove gases from the cooling system, some of which are designed to remove the non-condensable gases and avoid removing the vapor-phase coolant. A control system activates the venting system to achieve a desired pressure, which may be based on measured process conditions within the cooling system. A venting and refilling system may serve multiple cooling systems in a parallel arrangement.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of co-pending U.S.application Ser. No. 11/384,195, filed Mar. 17, 2006, which claims thebenefit of U.S. Provisional Application No. 60/775,496, filed Feb. 21,2006. Each of the foregoing applications is incorporated by reference inits entirety.

BACKGROUND

1. Field of the Invention

This invention relates generally to two-phase liquid cooling systems,and more particularly to venting and refilling multiple two-phase liquidcooling systems, such as those configured to cool rack mountedelectronics.

2. Background of the Invention

Liquid cooling is well known in the art of cooling electronics. As aircooling heat sinks continue to be pushed to new performance levels, sohas their cost, complexity, and weight. Because computer powerconsumptions will continue to increase, liquid cooling systems willprovide significant advantages to computer manufacturers and electronicsystem providers.

Liquid cooling technologies use a cooling fluid for removing heat froman electronic component. Liquids can hold more heat and transfer heat ata rate many times that of air. Single-phase liquid cooling systems placea liquid in thermal contact with the component to be cooled. With thesesystems, the cooling fluid absorbs heat as sensible energy. Other liquidcooling systems, such as spray cooling, are two-phase processes. In thetwo-phase cooling systems, heat is absorbed by the cooling fluidprimarily through latent energy gains. Two-phase cooling, commonlyreferred to as evaporative cooling, allows for more efficient, morecompact, and higher performing liquid cooling systems than systems basedon single-phase cooling.

An example two-phase cooling method is spray cooling. Spray cooling usesa pump to supply fluid to one or more nozzles, which transform thecoolant supply into droplets. These droplets impinge the surface of thecomponent to be cooled and can create a thin coolant film. Energy istransferred from the surface of the component to the thin-film ofcoolant. Because the fluid is dispensed at or near its saturation point,the absorbed heat causes the thin-film to turn to vapor. This vapor isthen removed from the component, condensed (often by means of a heatexchanger or condenser), and returned to the pump.

Significant efforts have been expended in the development andoptimization of spray cooling. A doctorial dissertation by Tiltonentitled “Spray Cooling” (1989), available through the University ofKentucky library system, describes how optimization of spray coolingsystem parameters, such as droplet size, distribution, and momentum cancreate a thin coolant film capable of absorbing high heat fluxes. Inaddition to the system parameters described by the Tilton dissertation,U.S. Pat. No. 5,220,804 provides a method of increasing a spray coolingsystem's ability to remove heat. The '804 patent describes a method ofmanaging system vapor that further thins the coolant film, whichincreases evaporation, improves convective heat transfer, and improvesliquid and vapor reclaim.

Dielectric fluids such as FLUORINERT® (a trademark of 3M Company) arewell-suited for use in electronic cooling systems, as they are safe forelectronic components and systems. The fluids have boiling points closeto atmospheric conditions and have latent heat of vaporization valuesthat provide efficient two-phase cooling.

A significant challenge in the use of some two-phase cooling systems ispresented by non-condensable gases. Dielectric fluids like FLUORINERT®can contain significant amounts of air and other non-condensable gasesin solution. When the dielectric fluid is placed into a system atatmospheric conditions, the fluid may thus contain a significant amountof air dissolved in the fluid. During use within a thermal managementsystem, according to Henry's Law, as the fluid approaches its saturationtemperature the amount of air in solution decreases. The air that waspreviously in solution occupies a volume within the system. According tothe ideal gas law, the partial pressure of the air will raise theboiling point of the cooling fluid above the natural saturation curve.This, in turn, reduces the performance of the cooling system because itraises the boiling point of the fluid above an optimal value. But someamount of air is useful for some cooling systems, for example, to avoidpump cavitation. The actual amount of air in the system can vary as airseeps into the system during operation, so it can be difficult tomaintain the amount of air within the system at an optimal level.

For the foregoing reasons, there is a need for a two-phase liquidcooling solution that can maintain an ideal amount of air or othernon-condensable gas within the system. With changing conditions inside acooling system, there is a need for a method of regulating thenon-condensable gases in the cooling system. Such a cooling system wouldresult in significant improvements in both the performance andreliability of the two-phase liquid cooling process.

SUMMARY OF THE INVENTION

To avoid at least some of the problems encountered with existingtwo-phase liquid cooling systems, as described above, a cooling systemwith active venting is provided. An active venting system activelyregulates the pressure within the cooling system, for example, byregulating the amount of non-condensable gases in the cooling system.With appropriate control of the active venting system, the performanceand reliability of the system can be increased and maintained over longand continuous periods of operation.

Embodiments of the invention include liquid cooling systems and methodsthat can provide thermal management for one or more electroniccomponents. In one embodiment, a cooling system includes a coolingliquid, or coolant, that is recirculated through a closed loop by one ormore pumps. The cooling fluid enters one or more cooling modules as aliquid or saturated liquid, and changes phase in the cooling module bymeans of latent energy gains. The resulting liquid and vapor mixture isthen removed from the cooling module and condensed so that it can bereturned to the pump and recirculated through the system. An activeventing system is coupled to a volume in the cooling system to regulatethe pressure in the cooling system. A control system is coupled to theactive venting system to activate the venting based on any of a numberof criteria, such as process conditions within the cooling system.

The active cooling system can exhaust gases out of the system usingvarious mechanisms. In one embodiment, a vent is located between thecooling module and the pump, and an auxiliary pump is coupled to thevent to pump a desired amount of gas out of the system. The controlsystem is coupled to the auxiliary pump and vent to provide the abilityto regulate the amount of gas removed from the system. The activeventing system may also be capable of adding gases into the coolingsystem (e.g., by pumping air into the system) when a pressure increaseis desired. In this way, the cooling system can regulate the pressure inthe cooling system to achieve a desired overall cooling efficiency.Adding air to the cooling system may also help avoid cavitation in thepumps.

Within the cooling system there may be one or more non-condensablegases. Non-condensable gases may include any gases or mixtures thereofthat do not condense into liquid form under conditions experiencedduring normal operation of the two-phase liquid cooling system. Air is acommon non-condensable gas in cooling systems, since they are typicallyrun at pressures below atmospheric so that air tends to seep in slowlythrough points in the system that are not completely sealed or otherwiseallow air permeation into the system. The non-condensable gases cause apartial pressure within the closed volume of the cooling system, whichalters the boiling point of the cooling fluid and thus affects theoperation of the cooling system. While removing the gases from thesystem, it is often desirable to remove the non-condensable gases whileminimizing the removal of the coolant in vapor phase. Otherwise, overtime the cooling system would lose coolant and would need to have thecoolant replaced. By removing non-condensable gases rather thanvapor-phase coolant from the cooling system, the need to replace coolantis reduced. Accordingly, the active venting system may be configured toremove an amount of the non-condensable gases from the system.

Various embodiments of the system include mechanisms in the ventingsystem for separating the coolant vapor from the non-condensable gasesto be removed. By separating the coolant vapor from the non-condensablegases, the active venting system can remove only the non-condensablegases and allow the coolant vapor to recycle through the cooling system.In one embodiment, the active cooling system includes using asemi-permeable membrane separator coupled between the cooling module andthe return line, allowing only coolant vapor to recycle through thesystem. In other embodiments, the active venting system comprises acondensing separator, a centrifugal gas separator, or a semi-permeablemembrane separator (such as a permeable tube vacuum system) to separatethe vapor cooling fluid from the non-condensable gases to be removed.

In one embodiment, the control system measures process conditions suchas the temperature and pressure within a volume of the cooling system.Based on the measured temperature and pressure, the control systemdetermines whether the process conditions within the system are inside adesired range. In one embodiment, the control system determines thatremoval of non-condensable gases is needed based on the saturation curveof the coolant. For example, the control system may detect when thepressure and temperature inside the system deviate from the saturationcurve of the coolant by a predetermined amount. When the control systemdetermines that venting is needed, it activates the venting system, forexample, causing the active venting system to open the vent and turn onthe auxiliary pump to remove gases in the system.

Multiple liquid cooling systems can also be vented and/or refilled in aparallel arrangement. In one embodiment of such a system, a centralsystem is coupled to the exhaust paths of a number of liquid coolingsystems. The central system may be selectably coupled to the exhaustpaths by controllable valves. When desired, the system removes exhaustgases from one or more of the cooling systems and vents the exhaustgases. Condensed coolant may be collected, and the system may supplycoolant liquid to one or more of the cooling systems via input ports,which may be coupled to the refilling system via controllable valves.When applied to multiple parallel cooling systems, such as multipleracks of liquid-cooled servers found in a data center, this embodimentallows for the automated venting and refilling of an entire system froma centralized location.

These and other features, aspects, and advantages of various embodimentsof the invention will become better understood with regard to thefollowing description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the course of the detailed description to follow, reference will bemade to the attached drawings. These drawings show different aspects ofembodiments of the present invention and, where appropriate, referencenumerals illustrating like structures, components, and/or elements indifferent figures are labeled similarly. It is understood that variouscombinations of the structures, components, and/or elements other thanthose specifically shown are contemplated and within the scope of thepresent invention:

FIG. 1 is a schematic diagram of a two-phase liquid cooling system withactive venting, in accordance with an embodiment of the invention.

FIG. 2 is a schematic diagram of a rack-mounted spray cooling system, inaccordance with an embodiment of the invention.

FIG. 3 is a schematic diagram of a semi-permeable membrane separator, inaccordance with an embodiment of the invention.

FIG. 4 is a schematic diagram of a condensing separator, in accordancewith an embodiment of the invention.

FIG. 5 is a schematic diagram of a centrifugal separator, in accordancewith an embodiment of the invention.

FIG. 6 is a schematic diagram of a permeable tube vacuum mechanism, inaccordance with an embodiment of the invention.

FIG. 7 is a chart showing a typical saturation curve for an examplecooling liquid.

FIG. 8 is a flow diagram of a control process for activating the activeventing system to remove non-condensable gases from the cooling system,in accordance with an embodiment of the invention.

FIG. 9 is a schematic of a venting and refilling system for servicingmultiple liquid cooling systems, in accordance with an embodiment of theinvention.

FIG. 10 is a chart showing the process conditions during operation of aliquid cooling system, in accordance with an embodiment of theinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Two-Phase CoolingSystem with Active Venting

FIG. 1 illustrates one embodiment of a two-phase liquid cooling system100 with active venting capabilities. The liquid cooling system 100includes at least one cooling module 105, a pump 110, a reservoir 115,and a condenser 120. The pump 110 pressurizes a supply of liquid coolantfrom the reservoir 115 and delivers the liquid coolant to the coolingmodule 105. The cooling module 105 places the liquid coolant in thermalcontact with a heat-producing device (not shown), such as but notlimited to computer processors, blade servers, circuit boards, memory,video cards, power devices, and the like. In the cooling module 105,heat from the heat-producing device transforms at least a portion of theliquid coolant into a vapor phase fluid. The cooling fluid istransferred to a condenser 120, which removes heat and condenses thevapor phase fluid back into the liquid phase and delivers it to areservoir 115. The liquid coolant can then be recycled in the system bythe pump 110.

Although the two-phase liquid cooling system 100 is shown with only themain components, the system 100 may include other well known components,such as filters, heaters, manifolds, coolers, and other components offluid systems. In addition, the system 100 is described as just oneexample of a system in which the active venting techniques describedherein can be applied. The system 100 may be a modular cold plate typesystem or a global cooling system where the cooling fluid comes directlyin contact with the electronics to be cooled. Moreover, the coolingsystem 100 is not limited to any particular type of two-phase liquidcooling system. Rather, the techniques described herein can be appliedto any type of two-phase liquid cooling system, such as, but not limitedto, spray cooling, micro-channels, mini-channels, pool boiling,immersion cooling, or jet impingement. Examples of liquid coolingsystems and their components that can be used with embodiments of theinvention are described in the following, each of which is incorporatedby reference in its entirety: U.S. Pat. No. 6,889,515, which describes aspray cooling system; U.S. Pat. No. 6,955,062, which describes a spraycooling system for transverse thin-film evaporative spray cooling; andU.S. Pat. No. 5,220,804, which describes a high heat flux evaporativespray cooling; and U.S. Pat. No. 5,880,931 which describes a spraycooled circuit card cage.

Coupled to the cooling system 100 is an active venting system 125 forremoving gases and/or adding gases to the liquid cooling system 100. Asshown in FIG. 1, the active venting system 125 may be coupled to avolume in the system 100 where gases are present, such as the volumeabove liquid coolant in the reservoir 115. In other embodiments, theventing system 125 may be coupled to other places in the flow path ofthe cooling system 100, such as in a return manifold in the path fromthe cooling module 105 to the pump 110 (such as return manifold 240shown in FIG. 2, for example). In the embodiment shown in FIG. 1, theventing system 125 comprises an auxiliary pump 130 coupled to the volumein the reservoir 115. The auxiliary pump 130 is further coupled to acheck valve 135, which prevents air from entering the venting system125.

A control system 140 is coupled to the venting system 125 to provide forselective activation of the venting of gases by the venting system 125.Using control signals (illustrated as dotted lines in FIG. 1), thecontrol system 140 may control the auxiliary pump 130, thereby causingthe venting system 125 to remove and/or add gases into or out of thecooling system 100. For other embodiments of the venting system 125, thecontrol system 140 is configured to provide appropriate control signals.

In one embodiment, the non-condensable gases removed by the activeventing system 125 are released into the surrounding environment. Insome applications, however, it is undesirable to allow thenon-condensable gases to be released. To address this need, in anotherembodiment, the active venting system 125 vents, pumps, or otherwisedirects the non-condensable gases removed from the cooling system 100into a sealed chamber 160 for storage therein. The sealed chamber allowsthe cooling system 100 to be used in very sensitive areas where thenon-condensable gases cannot be introduced.

In another embodiment, the gas storage chamber 160 houses a condenserunit 162, which may comprise condensing fins that aid in condensing anyvapor in the chamber 160. The chamber is further coupled to a reliefvalve 164. The relief valve 164 is designed to relieve the stored orcollected non-condensable gases once a certain pressure inside thestorage chamber 160 is reached. In one embodiment, the pressure reliefvalve 164 comprises a spring-loaded valve that automatically opens at a10 psi differential between the inside of the chamber and theatmosphere. With the chamber 160 at room temperature, the added pressurehelps to ensure that only air escapes from the system.

The control system 140 activates the venting system 125 based on processconditions within the cooling system. In this way, the control system140 can achieve certain desired operating conditions in the coolingsystem 100. Although a variety of process conditions can be used todescribe the cooling system, in one embodiment the process conditionsinclude the pressure and temperature of the gases above the liquidcoolant in the reservoir 115. Accordingly, a pressure transducer 145 andtemperature sensor 150 (which may comprise a thermocouple, thermistor,resistance temperature detector (RTD), thermopile, infrared sensor, orany other suitable temperature sensor) coupled to the reservoir 115provide readings of these process conditions. The control system 140uses these pressure and temperature readings to determine whether andwhen to activate the venting system 125. Various embodiments ofalgorithms that the control system 140 can be used to activate theventing system 125 are described in more detail below; however, it canbe appreciated that the control system 140 can receive additional typesof inputs and can be programmed to perform any number of algorithms toachieve a desired effect in the cooling system 100. Moreover, thepressure transducer 145 and temperature sensor 150 may be located atother parts of the system, such as in a return manifold path 240.

Although the control system 140 is illustrated as a separate system inFIG. 1, it can be integrated into the active venting system 125 or anyother part of the cooling system 100. Moreover, the control system maybe implemented, in whole or in part, by hardware, software, firmware, ora combination thereof.

The active venting techniques described herein can be implemented invarious types of two-phase liquid cooling systems. For example, FIG. 2schematically illustrates a rack-mounted spray cooling system in whichan embodiment of the active venting technique is employed. As shown inFIG. 2, a pump 210 directs a coolant through a plurality of spraycooling modules 220. Each spray cooling module 220 is located in arack-mounted device and is configured to cool one or more heat-producingelectronic devices by spraying the coolant liquid on the devices or on asurface thermally coupled thereto. The resulting two-phase coolant isthen returned to a thermal management unit 250 by way of a returnmanifold 240. The two-phase coolant is condensed in the return manifold240 and/or in the thermal management unit 250, where the liquid coolantis stored until being recycled through the system by the pump 210.

An active venting system 260 is coupled to the return manifold 240,where it has access to gases in the flow path of the cooling system. Asdescribed above, the active venting system 260 may remove gases fromand/or add gases to the flow path of the cooling system to adjust thepressure therein and thus affect the operation of the cooling system.Rather than being coupled to the return manifold 240, the active ventingsystem 260 may alternatively be fluidly coupled to a volume of gas inthe thermal management unit 250 for exchanging gases therewith. In arack-mounted cooling system, the active venting system 260 and thethermal management unit 250 may also be rack-mounted devices.

One problem with the startup of a rack-mounted spray cooling system,where the supply manifold 230 and the return manifold 240 are mounted inthe rack vertically, is that air can become trapped in the supplymanifold 230 above the uppermost connection that leads to the uppermostcooling module 220. The trapped air undesirably increases systempressure, and because there is no fluid flow above the uppermostconnection, the non-condensable gases must dissolve back into thecoolant to be removed. It has been shown to take several days for thenon-condensable gases to be removed fully with this configuration. Aftera system is shut down, moreover, a substantial amount of non-condensablegas may collect in the supply manifold 230, which again takessignificant time to remove.

To address this problem, in one embodiment, a bypass flow path 270 isplaced between the supply manifold 230 and the return manifold 240 nearthe tops thereof. The flow path 270 allows a small flow (e.g., around 1%of the full flow) of gas to pass from the top of the supply manifold 230to the return manifold 240. The flow path 270 may comprises a tube, anda chemical filter 265 may be installed in the flow path 270, since thisprovides an ideal service location. The bypass flow path 270 withchemical filter 265 could replace a bypass filtration line that is oftenused within the thermal management unit 250. In an alternativeembodiment, the bypass path 270 can be separate from the filter 265,although it is typically desired to reduce number of fluid joints in thesystem.

Active Venting System Embodiments

As described above, many coolants used in two-phase fluid coolingapplications may absorb a significant amount of air or othernon-condensable gases. Because the non-condensable gases remain in gasform throughout the cooling system, they impart a partial pressure thatadds to the pressure within the cooling system. Although a slightlyincreased pressure may be useful to avoid cavitation in the pumps, itcan also have detrimental effects on the cooling performance of thesystem by increasing the boiling point of the coolant. Accordingly, itis often preferable to control the amount of non-condensable gases thatare present in the cooling system. When removing gases from the system,therefore, it is generally preferable to remove the non-condensablegases while leaving the coolant vapor in the system. Various embodimentsof the active venting system designed to achieve this purpose aredescribed below.

FIG. 3 depicts a semi-permeable membrane separator embodiment forfacilitating removal of non-condensable gases by a venting system. Thisembodiment is described in the context of the cooling system of FIG. 2,but it could be employed in any other type of cooling system. Asillustrated, a semi-permeable membrane 310 may be located in a parallelconfiguration with the flow path between the return manifold 240 and areturn line 320 leading to a thermal management unit 250, or with someother portion of the flow path. The membrane 310 is designed to bepermeable to the coolant but not to the non-condensable gases that areexpected to be in the system.

During operation of the venting system, the side of the membrane 310that includes the coolant and non-condensable gas mixture is increasedin pressure (e.g., by a pump, not shown). In this way, the coolant isallowed to pass through the membrane 310 and return to the thermalmanagement unit 250, while the non-condensable gas remains in themanifold 240 (or another volume from which the venting system canextract gas). This increases the concentration of the non-condensablegas versus the coolant vapor in the manifold 240. If the venting systemtakes gases from the manifold 240, the gas mixture taken by the ventingsystem will thus have a relatively higher concentration ofnon-condensable gas versus coolant vapor than in the rest of the system.In another embodiment, the membrane 310 can be configured in the reversemanner (such as in the embodiment described below in connection withFIG. 6).

FIG. 4 shows a condensing separator embodiment of an active ventingsystem 410. This embodiment of the venting system 410 is designed toreceive coolant vapor air mixture from the cooling modules, e.g., bytapping into the return manifold 240 of a cooling system such as thatshown in FIG. 2. The venting system 410 could tap into the flow path ofthe cooling system downstream of a condenser or in a reservoir of a heatexchanger, but there would be less need for the condensing function ofthis embodiment since the coolant would be expected to be primarily inthe liquid phase in those areas of the cooling system.

In operation, the venting system 410 receives a mixture of the coolantvapor and non-condensable gases from the return manifold 240. A valve425 may be provided on the gas input line 420 to control when theventing system can take in the gases. The input gases are received in achamber of the venting system 410, where a condenser 430 reduces thetemperature of the gases until the coolant vapor condenses and collectsas a liquid in the venting system. When a control system determines thatthe venting system should be activated to expel non-condensable gas fromthe system, the control system activates an auxiliary pump (as shown inFIG. 1) or other mechanism for removing some amount of thenon-condensable gas in the venting system 410 through an exit port 460.The control system may cause the input valve 425 to close for a periodof time before activating the auxiliary pump, thereby giving thecondenser 430 sufficient time to condense the coolant vapor to ensurethat most of the gas expelled is the non-condensable gas.

At various times, such as when the venting system has a predeterminedamount of liquid coolant collected (e.g., as measured by a level sensor,not shown), a liquid return pump 440 is activated. The liquid returnpump 440 passes the condensed liquid coolant from the venting system 410back to the return manifold 240 by way of a liquid return line 450. Aliquid return valve 455 may be provided in the liquid return line 450 toprevent liquid coolant from backing up into the venting system 410. Inthis way, the coolant vapor from the cooling modules is condensed sothat it can be recycled through the system, rather than being ventingfrom it. The pump 440 may be optional, e.g., the coolant may be gravitydrained from the reservoir and reintroduced into the cooling system aswell.

FIG. 5 illustrates a centrifugal separator embodiment of an activeventing system 510, which separates the coolant vapor from thenon-condensable gases. As illustrated, the venting system 510 may tapinto the return manifold 240 of a cooling system such as that shown inFIG. 2; however, as with the condensing separator embodiment 410, theventing system 510 could tap into other points in the flow path of thecooling system. The active venting system 510 thus receives a mixture ofcoolant vapor and non-condensable gases in a gas input line 530, whichmay be opened or closed using an input valve 535. The received mixtureof gases is provided to a centrifugal vapor pump 520, which is designedto separate the coolant vapor and non-condensable gas based on thedifference in their densities.

The centrifugal vapor pump 520 is activated by the control system whenit is determined that the venting system 510 should remove gas from thecooling system. The centrifugal vapor pump 520 removes dissolvednon-condensable gas from the coolant vapor by passing the mixed gasstream through a series of rapidly spinning disks. As the rotationalmotion is imparted to the gas stream, the more dense gases (e.g.,FLUORINERT®, in a mixture of FLUORINERT® and air) are forced to theperimeter, while the less dense gases continue down the center of thedevice and exit the centrifugal pump. The centrifugal vapor pump 520 canbe controlled by manipulating the rotation speed of the spinning disksby an ordinary brushless DC controller, and by the flow rate of thevacuum pump that pulls the mixed vapor through the device and vents tothe atmosphere. Alternatively, where the coolant vapor is less densethan the non-condensable gases, the configuration may be changed toallow the denser gases to be removed.

In the embodiment shown in FIG. 5, the venting system 510 is designedfor a cooling system in which the non-condensable gases are less densethan the coolant vapor. The non-condensable gases are expelled from theventing system 510 via a line 550 and through an exhaust port 555, whichpreferably does not allow air to pass into the venting system 510. Thedenser coolant vapor returns to the return manifold 240 in a coolantreturn line 540. The coolant return line 540 may include a valve 545 toprevent coolant from entering the venting system 510 through the returnline 540.

FIG. 6 shows another embodiment of a venting system 610 for removingnon-condensable gases from a closed-loop cooling system. In thisembodiment, at least a portion of the return path of the cooling systemis passed through a coil or bundle of semi-permeable tubing 620, whichis permeable to non-condensable gases but not permeable to the coolant.(Although FIG. 6 shows a short length of tubing 620 in the housing 630,having a coil or bundle of tubing 620 with a long length relative to thediameter of the tubing 620 increases the ratio of surface area tovolume, thereby facilitating removal of non-condensable gases from thesystem.) In one embodiment, the coolant is FLUORINERT® and tube 620 isimpermeable to FLUORINERT® but does exhibit marked permeability to air.The tubing 620 is located in a sealed housing 630, which is coupled to avacuum pump 640 by tubing 650 that is not permeable. When the vacuumpump 640 is activated, a vacuum is applied to the inside of the housing630, and thus, to the outside of the semi-permeable tubing 620. Thiscauses the non-condensable gas to migrate through the tubing 620, whilethe coolant is left inside the tubing 620. The non-condensable gas isexpelled from the housing 630 by the vacuum pump 640 through an exhaustline 660. The coolant, on the other hand, continues through the tubing620 and is returned to the cooling system to be recycled.

In one embodiment, the tubing 620 comprises a co-extrusion having two ormore layers, although the tubing 620 need not necessarily have more thanone layer. In a multilayer embodiment, an exterior layer of theco-extruded tubing 620 may comprise ether or ester-based polyurethane,which is appropriate due to its high air and low PFC permeationproperties. An interior layer of the co-extruded tubing 620 may comprisepolyethylene, which has excellent fluid compatibility properties. Thetubing 620 is preferably a semi-permeable membrane. This is in contrastto the tubing used in other parts of embodiments of the cooling system,in which co-extruded tubing that prevents permeation and provides goodfluid compatibility while remaining flexible is used.

On one embodiment, the tubing used for some or all flexible connectionswithin the system is a co-extruded tubing that comprises:

-   -   an outer layer composed of an Engage 8440 with Ampshield 1199:        Ethylene Octene Co-polymer, where Ampshield is a 52% flame        retardant in a low-density polyethylene carrier (0.032″ thick);    -   a binding layer comprised of Bynel 4157: Linear low density        polyethylene (LLDPE) (0.003″ thick);    -   a next layer EVALCA F101: ethyl vinyl alcohol (EVOH) (0.005″        thick);    -   a next binding layer of Bynel 4157: Linear low density        polyethylene (LLDPE) (0.003″ thick); and    -   an inner layer of Engage 8440: Ethylene Octene Co-polymer (0.02″        thick).        The two Bynel layers in the above construction are binding or        “tie” layers. The innermost layer is not adversely affected by        fluids common to liquid cooling of electronics. The innermost        layer also remains highly flexible at structural thicknesses and        environmental conditions typically found for rack-mounted        products. The EVOH layer is impermeable to FC-72, PF-5060, and        other fluids commonly used in the electronics cooling industry,        as well as to air or non-condensable gases. But because the EVOH        layer tends to be too stiff if implemented in greater        thicknesses, it is impractical as a flexible tubing by itself.        Its presence in the co-extrusion is to prevent cooling fluid        and/or air permeation, while its minimal thickness does little        to affect flexibility. The outermost layer adds structural        integrity without adversely affecting flexibility. In various        embodiments of the system, a commercially available version        containing a flame retardant may be chosen due to its enhanced        commercial viability in the marketplace. Other co-extrusions        that implement a different layer order may be used, as well as        fewer layers or different thicknesses of the layers, although        stiffness or permeability may be sacrificed with variations. To        increase the flexibility of the tubing temporarily (e.g., to        remove stresses in an installed system), the tubing can be        heated.

Alternatively, the venting system could be designed using a tubing thatis permeable to the coolant but not to the non-condensable gas. In sucha case, the tubing could comprise polyvinylidine fluoride (PVDF), orKYNAR®, which is permeable to FLUORINERT® but not to air. The coolantwould be collected outside of the tubing and returned to the system,while the non-condensable gas left in the tubing would be exhausted fromthe system.

Embodiments of the co-extruded tubing described herein may be used forall fluid connections in the system where rigid tubing is impractical,such as to connect pumps to the supply manifold, the supply manifold tothe spray modules, the spray modules to the return manifold, and thereturn manifold to the condenser. The co-extruded tubing may alsoconnect the active venting system to the return manifold. The selectionof materials for this and any other tubing may depend, in part, on thetype of coolant used.

Operation

Controlling the pressure inside of the cooling system may be vitallyimportant for many applications, as demonstrated by the saturation curveplotted in FIG. 7. The saturation curve provides the boiling point for aparticular coolant for a range of pressures. It is often desirable tooperate above the coolant's saturation curve, since the pumps cancavitate if the pressure is too low for a given temperature ofoperation. Adding air or other non-condensable gases is one way to moveabove the saturation curve to allow the pumps to operate. But with toomuch air the coolant evaporates at a relatively high temperature, whichcauses the two-phase cooling modules to operate at a higher temperature.Accordingly, in one embodiment, the cooling system includes an amount ofair or other non-condensable gas in the system to balance thesecompeting concerns. This is illustrated by the “ideal operatingcondition” curve in FIG. 7, although what is considered ideal operatingconditions may change from application to application, so the curve inFIG. 7 is presented for illustration purposes only.

In one embodiment, the cooling system can regulate the amount ofnon-condensable gases in the cooling system using a control algorithmimplemented by the control module described above. FIG. 8 provides oneembodiment of a control algorithm for maintaining the cooling system ator near an ideal operating curve. In this control process, the pressuretransducer 145 and temperature sensor 150 measure 810 the pressure andtemperature in a location in the cooling system (such as a volume over areservoir or a point in the return path). Based on this measuredpressure and temperature, the control system calculates the idealoperating pressure of the system for the measured temperature. If 820the difference between this ideal pressure and the saturation curve atthe measured temperature is above a predetermined maximum differential(e.g., 3 psi), the control system activates 830 the venting system toreduce the pressure in the cooling system. Otherwise, the control systemturns or keeps 850 off the venting system, after which the pressure andtemperature are measured 810 again in a subsequent interval.

In one embodiment, the control system activates 830 the venting systemaccording to a predetermined profile, which specifies an amount of timeon and off for the venting system. The on period of the profile allowsthe system to exhaust a non-condensable gas for a period of time, whilethe off period allows the venting system to separate the coolant vaporfrom the non-condensable gas. The off period also allows the system as awhole to come into equilibrium, while other entrained non-condensablegases are moved to the venting system so they can be extracted. Althoughthe particular profile used may depend on the system parameters, in oneembodiment the profile is 10 seconds of venting followed by 3 minutesoff. The venting system runs (e.g., according to the profile) until thecontrol system determines 840 that the system pressure is within apredetermined differential (e.g., 2.5 psi) of the saturation curve atthe system temperature. Once this condition is met, the control systemturns 850 the venting system off, and the control cycle repeats.

In one embodiment, the control system may check the pressure differencebetween the system and the saturation curve so that it can maintain thesystem above a minimum differential (e.g., a 1.8 psi). This checking mayoccur, for example, continually during the running of a profile for theventing system. If the cooling system does come within the predeterminedminimum differential of the saturation curve, the control systemautomatically shuts the venting system off. This helps to prevent thepumps from cavitating due to too low of a pressure in the coolingsystem.

During startup of the cooling system there may be different ventingneeds than during normal operation. For example, there is typically moreneed for venting since there is more air that has seeped into thecooling system. Moreover, the system can tolerate faster venting becausethe system is stagnant at startup; therefore, the vapor and air are moreseparated from one another. Once fluid is pumped through the system, theair and vapor tend to mix and extraction has to be done more slowly.Accordingly, a startup profile may be run until the cooling systemreaches a desired point from the saturation curve, where the startupprofile has more aggressive venting than the regular profile. In oneembodiment, the startup profile runs the venting system for 55 secondson and 5 seconds off, for up to 5 minutes or until the cooling systemreaches 5 psi above the saturation curve. As with the regular ventingprofile, various other startup profiles may be defined based on othersystem parameters and needs.

Rather than trying to maintain the cooling system at an ideal operatingcurve, the control system can also be used to maintain the coolingsystem at a given temperature. This may be useful, for example, as atool for the testing or burn-in of semiconductors. Becausenon-condensable gases within the working fluid of the system affect thecomponent temperatures, adding the gases to the system or allowing thegases to remain in the system can raise the temperature of thecomponents being cooled by the system. The control system may thereforereceive additional inputs, such as the temperature of a particularcomponent attached to the cooling system. By adjusting the gases withinthe cooling system, the control system can maintain these inputs atdesired values.

Venting and Refilling of Multiple Cooling Systems

In other embodiments, the automated venting of two-phase liquid coolingsystems can be applied by a central system coupled to a plurality ofcooling systems in a parallel arrangement. The cooling systems may berack-mounted cooling systems, where the overall system extends theautomated venting techniques described herein to the multi-rack level.In addition to automated venting of multiple cooling systems,embodiments of the invention can also provide for the automated refillof multiple cooling systems from a central reservoir.

FIG. 9 illustrates one embodiment of a centralized system for providingventing and refilling for multiple two-phase liquid cooling systems 910.Each cooling system 910 may comprise a plurality of two-phase liquidcooling modules, such as those described above. The plurality of coolingmodules may further be configured within electronic devices arranged ina rack-mounted system, as found in computer server environments, or theymay be any other type of two-phase liquid cooling systems for whichventing and/or refilling are desired. The components of the centralizedventing and refilling system are coupled to the cooling systems 910 in aparallel arrangement. This arrangement allows the various liquid coolingsystems 910 to be vented and/or refilled concurrently, and it alsoallows a subset of the cooling systems 910 to be serviced at any onetime by closing the corresponding fluid path, as described below.

In the embodiment shown, the centralized venting and refilling systemcomprises a compressor 930, a separation column 940, a reservoir 950, anexhaust valve 970, and a fill valve 980. The compressor 930 of thecentralized system is coupled to an exhaust path from each of the liquidcooling systems 910. The exhaust path for a particular cooling system910 may be coupled from a simple vent from a condenser in the coolingsystem 910, or it may be coupled to any other vent or port designed toexhaust gases from the cooling system 910. A vent valve 920 in eachexhaust path separates the compressor 930 from the corresponding coolingsystem 910. Each vent valve 920 can be opened and closed to control whengases are allowed to vent from the corresponding cooling system 910.

The centralized venting and refilling system may also include returnpaths that couple the system to an input port of each liquid coolingsystem 910. These return paths allow selective refilling of each of thecooling systems 910 with the coolant liquid from the reservoir 950. Arefill valve 960 couples the input port of each cooling system 910 tothe reservoir 950, thereby allowing control of when and how much eachcooling system 910 is refilled. In one embodiment, the reservoir 950 isheld at a higher pressure than each cooling system 910, so the coolantliquid in the reservoir 950 naturally flows into each cooling system 910when the corresponding refill valves 960 are opened. Alternatively, apump or other means may be used to cause the coolant liquid to flow fromthe reservoir 950 into the cooling systems 910 when desired. In oneembodiment, the vent valves 920 and/or refill valves 960 may beautomatically controllable valves, such as solenoid valves, whichfacilitate control of the valves' state from the centralized, automatedventing and refill system.

FIG. 10 illustrates the temperature and pressure conditions in theventing and refilling systems during operation of the system inaccordance with one embodiment. As with certain embodiments describedabove, the operation of this system is described using the coolantliquid PF-5060; however, it can be appreciated that embodiments of theinvention can be practiced with a variety of other coolant fluids, alsoas described above. In the example shown, the desired operating point ofan individual cooling system 910 (e.g., one rack in a multi-rack system)is identified on the PF 5060 saturation curve as point 1. It is oftendesirable to operate the cooling system 910 at a sub-atmosphericpressure to provide lower possible CPU temperatures. Operating atsub-atmospheric pressures also tends to cause any leakage that occurs inthe system to be air ingress only, thereby avoiding coolant vapor escapeinto the atmosphere.

During normal operation of the cooling systems 910, air will typicallyleak into the system. This may occur due to any servicing operation(e.g., when the servers containing the individual cooling modules areattached to the cooling system's manifold), during servicing of thethermal management units of each cooling system 910, or from permeationor minor leaks through seals and joints in the gas lines. As air leaksinto each cooling system 910, the pressure climbs towards point 2 on thePF-5060 saturation curve. As the saturation pressure in the coolingmodule increases with the overall system pressure, the temperature ofthe cooling module—and hence, of a CPU or other device beingcooled—likewise increases.

When the automated venting system determines that the pressure ortemperature has reached a predetermined maximum (e.g., at point 2 on thechart of FIG. 10), the vent valves 920 for the cooling systems 910 areopened. The system may open all of the vent valves 920 at this time, orit may open only a subset of the vent valves 920 corresponding to thecooling systems 910 that are determined to need venting. This latterembodiment may be useful when the cooling systems 910 are monitoredindividually, and the system may determine that some, but not all, ofthe cooling systems 910 need venting to reduce the pressures therein.

Once the desired vent valves 920 are opened, the compressor 930 isturned on while the exhaust valve 970 is kept shut. This causesair—which may accumulate in the top of a vent manifold or condenser ofthe cooling system—along with a small amount of coolant vapor to bepumped into the separation column 940 from the cooling systems 910. Thecompressor 930 may continue to pump until the separation column 940exceeds a predetermined amount, such as 20 psi. The vent valves 920 arethen closed and the compressor 930 is turned off, trapping an amount ofair and coolant vapor at the increased pressure.

The gases and fluid in the separation column 940 are then allowed tocool, e.g., to room temperature (or about 20° C.). At this point, thesystem is at point 3 on the chart of FIG. 10, where the saturationpressure of the PF-5060 coolant vapor is around 4 psi, and the remainingpressure in the column 940 is due to air and any other gasses in thesystem. The coolant vapor in the separation column 940 will tend tostratify, as PF-5060 vapor is more than ten times denser than the air inthe column 940. The exhaust valve 970 near the top of the separationcolumn 940 is then opened, which causes gases to vent from the column940 due to the increased pressure in the column 940. Because the air ismostly at the top of the column 940, the majority of the gases beingvented will tend to be air, thus limiting the amount of coolant vaporlost from the system. Alternatively, any one or any combination of theseparation methods described herein may be used in lieu of or incombination with the separation column 940.

Over time, and from venting multiple cooling systems 910, coolant liquidwill tend to accumulate in the system. In one embodiment, the separationcolumn is coupled to the reservoir, which receives condensed coolantliquid from the column 940. This coolant liquid can be used to refillany individual cooling system 910 that is running low by simply openingthe refill valve 960 for the corresponding cooling system 910. Anincreased pressure in the reservoir 950, or an optional pump, may beused to cause the coolant liquid to flow from the reservoir 950 into thedesired cooling system 910. As needed, coolant fluid can be added to thesystem centrally by adding coolant fluid to the reservoir 950 via thefill valve 980.

As described herein, the entire system may be automated to ventindividual or multiple cooling systems 910 based on the pressure in thecorresponding cooling systems 910. The system may also be automated tofill the cooling systems 910 as needed based on the cooling liquid levelin each cooling system 910. Embodiments of the system may be applied torack-mounted systems, in which each rack containing multiple servers orother computer systems is treated as a single cooling system. The racksare then vented and refilled using the central system, thus allowing forscaling of the cooling systems without a proportional scaling of theassociated maintenance. The centralized system also allows forautomation, further reducing the maintenance of a multi-rack orotherwise plural cooling systems configuration. This system may thusgreatly improve the ease of servicing in large data centers.

SUMMARY

The foregoing description of the embodiments of the invention has beenpresented for the purpose of illustration; it is not intended to beexhaustive or to limit the invention to the precise forms disclosed. Forexample, many of the fastening, connection, manufacturing, and othermeans and components that are described in various embodiments arewidely known in the relevant field, and their exact nature or type isnot necessary for a person of ordinary skill in the art or science tounderstand the invention. Persons skilled in the relevant art canappreciate that many modifications and variations are possible in lightof the above teachings. It is therefore intended that the scope of theinvention be limited not by this detailed description, but rather by theclaims appended hereto.

1. A centralized system for venting and refilling a plurality oftwo-phase liquid cooling systems, each cooling system configured torecirculate a coolant through the cooling system and evaporatively coolone or more devices, the servicing system comprising: a ventinginterface for receiving vented gases from the plurality of coolingsystems, the venting interface adjustable to receive vented gases fromany one or combination of the cooling systems; a venting system coupledto the venting interface and configurable to exhaust at least some ofthe received vented gases from the cooling systems; a condenser coupledto the venting system for condensing at least a portion of a condensablecoolant in the received vented gases; a reservoir coupled to thecondenser to receive the condensed coolant; and a refill interfacecoupled to the reservoir for providing the condensed coolant to thecooling devices, the refill interface adjustable to provide condensedcoolant to any one or combination of the cooling systems.
 2. The systemof claim 1, wherein the venting interface comprises: a plurality oflines for coupling the venting system to vent ports of the coolingsystems in a parallel arrangement; and an adjustable valve in each ofthe lines to allow or prevent flow of vented gases in each line.
 3. Thesystem of claim 2, wherein the adjustable valves are automaticallycontrollable to allow or prevent flow of vented gases in each line. 4.The system of claim 2, wherein the adjustable valves are solenoidvalves.
 5. The system of claim 1, wherein the venting system comprises:a compressor for pumping gases from the venting interface; and anexhaust valve adjustable to allow or prevent flow of gases out of theventing system.
 6. The system of claim 5, wherein the venting system isconfigured to pump gases from the venting interface until at least apredetermined pressure is reached within one or more of the coolingsystems, and then open the exhaust valve to exhaust at least a portionof the gases from the venting system.
 7. The system of claim 1, whereinthe condenser comprises a condensing column arranged vertically tofacilitate separation of vented gases by density.
 8. The system of claim1, wherein the venting system comprises a semi-permeable membraneseparator for exhausting at least some of the received vented gases. 9.The system of claim 1, wherein the venting system comprises a condensingseparator for exhausting at least some of the received vented gases. 10.The system of claim 1, wherein the venting system comprises acentrifugal separator for exhausting at least some of the receivedvented gases.
 11. The system of claim 1, wherein the venting systemcomprises a permeable tube vacuum mechanism for exhausting at least someof the received vented gases.
 12. The system of claim 1, wherein thereservoir includes a fill interface to receive liquid coolant from anexternal source.
 13. The system of claim 1, wherein the refill interfacecomprises: a plurality of lines for coupling the reservoir to inputports of the cooling systems in a parallel arrangement; and anadjustable valve in each of the lines to allow or prevent flow of liquidcoolant in each line.
 14. The system of claim 13, wherein the adjustablevalves are solenoid valves.
 15. A multi-loop liquid cooling systemcomprising: a plurality of closed-loop, two-phase liquid cooling systemsconfigured to recirculate a coolant and evaporatively cool one or moredevices, each cooling system including an output port for venting gasesfrom the cooling system and an input port for receiving coolant; and acentral venting system coupled in a parallel arrangement to the outputports of each of the cooling systems to receive vented gases from thecooling systems, the venting system configured to condense an amount ofcoolant vapor in the received vented gases and to exhaust an amount ofnon-condensable gas in the received vented gases.
 16. The system ofclaim 15, wherein each of the cooling systems comprises a plurality ofrack-mounted cooling modules thermally coupled to cool a plurality ofrack-mounted computing devices.
 17. The system of claim 15, wherein theventing system is coupled to each of the cooling systems by a closeablevalve, each of the valves independently closeable.
 18. The system ofclaim 15, further comprising: a central reservoir coupled to the ventingsystem to receive the condensed coolant therefrom, the reservoir coupledin a parallel arrangement to the input ports of each of the coolingsystems to provide an amount of coolant thereto.
 19. The system of claim18, further comprising: a valve coupled between the central reservoirand each cooling system, each valve adjustable to allow or prevent flowof coolant to the corresponding cooling systems.
 20. The system of claim15, further comprising: a valve coupled between the venting system andeach cooling system, each valve adjustable to allow or prevent flow ofvented gases from the corresponding cooling systems.
 21. The system ofclaim 15, wherein the venting system comprises: a compressor for pumpingvented gases from the cooling systems; and an exhaust valve adjustableto allow or prevent flow of then vented gases out of the venting system.22. The system of claim 21, wherein the venting system is configured topump the vented gases from the venting interface until at least apredetermined pressure is reached within the venting system, and thenopen the exhaust valve to exhaust at least a portion of the vented gasesfrom the venting system.
 23. The system of claim 15, wherein thereservoir includes a fill interface to receive liquid coolant from anexternal source.
 24. The system of claim 15, wherein the reservoir iscoupled to each of the cooling systems by a closeable valve, each of thevalves independently closeable.
 25. A centralized system for servicingmultiple two-phase liquid cooling systems, the system comprising: ameans for selectively receiving an exhaust stream from each of aplurality of two-phase liquid cooling systems; and a means for ventingthe received exhaust streams.
 26. The system of claim 25, furthercomprising: a reservoir coupled to the means for venting to receivecondensed liquid coolant therefrom; and a means for selectivelyrefilling one or more of the cooling systems.
 27. A method for venting anon-condensable gas from a plurality of two-phase liquid coolingsystems, the method comprising: cooling one or more heat-producingdevices by recirculating a coolant through each of a plurality ofcooling systems and evaporatively cooling the heat-producing devicestherewith; venting gases from at least some of the cooling systems intoa central chamber; and exhausting at least a portion of the vented gasesfrom the central chamber.
 28. The method of claim 27, wherein each ofthe cooling systems comprises a plurality of rack-mounted coolingmodules thermally coupled to cool a plurality of rack-mounted computingdevices.
 29. The method of claim 27, further comprising: determiningwhether to vent each of the cooling systems based at least in part on aprocess condition within the cooling system; wherein the venting is inresponse to the determining.
 30. The method of claim 29, wherein theprocess condition is a pressure within the cooling system.
 31. Themethod of claim 27, wherein the venting comprises opening a gas linebetween the central chamber and each cooling system to be vented,pumping an amount of gas from each corresponding cooling system, andclosing each opened gas line.
 32. The method of claim 27, wherein theventing comprises pumping gases from at least one cooling system untilat least a predetermined pressure is reached within the cooling system.33. The method of claim 27, further comprising: condensing a portion ofthe vented gas to reform a liquid coolant in a central reservoir; andrefilling one or more of the cooling systems with the liquid coolant.34. The method of claim 33, wherein the refilling comprises selectivelyopening a fluid line from the central reservoir to one or more of thecooling systems.
 35. The method of claim 34, wherein the centralreservoir is held at a higher pressure than the cooling systems, therebycausing the liquid coolant to flow to the cooling systems.
 36. Themethod of claim 33, adding liquid coolant to the central reservoir froman external source.